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ABSTRACT 



Survey Results of Urban School Classroom Practices 
in Mathematics and Science : 2000 Report 

Using the Survey of Enacted Curriculum 
Conducted during Eight USI Site Visits 

The Survey of Enacted Curriculum (SEC) is being conducted as a study component of a three year 
grant from the National Science Foundation (NSF), entitled “How Reform Works: An Evaluative 
Study of NSF’s Urban Systemic Initiatives (USI).” The evaluative study is exploring the impact of USI 
programs on student achievement and the learning infrastructure in urban school districts. These results 
will be used to develop an inferential causal model linking the systemic reform drivers and other key 
elements. 

The purpose of the Survey of Enacted Curriculum is to analyze classroom practices and 
curriculum, and to assess the impact of USI and other reform efforts. In addition to classroom 
practices, the survey collects data regarding teachers preservice education and professional 
development experiences. Survey results from elementary and middle school teachers who 
received varying amounts of professional development (High PD and Low PD) will be compared. 

In brief, some survey findings are as follows: 

• About 80 to 90% of USI teachers were actively involved in professional develop- 
ment, which focused on content standards, in-depth study of content, curriculum 
implementation, multiple strategies for assessment, and new methods of teaching. 

Teachers report that as a result of professional development they are using and 
applying new methods and standards in classrooms (average 2.3 on scale of 1 to 3). 

• In science, teachers with High PD report greater use of multiple assessments than 
their counterparts with Low levels of professional development, especially at the 
elementary level. 

• For science and mathematics, state and district frameworks or standards have the 
greatest positive influence on curriculum, as well as national standards. Differences 
between High PD and Low PD teachers were not notable. 

The first report. Survey Results of Urban School Classroom Practices in Mathematics and 
Science: 1999 Report, presented a summary of the SEC from four USI sites surveyed in 1999; 
Baltimore, Dallas, Detroit, and Phoenix. In 2000 four additional USI sites were surveyed; 
Columbus, Fresno, Memphis, and Philadelphia. This report presents a summary of findings of 
both 1999 and 2000 survey results. 
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CHAPTER I 
INTRODUCTION 



1.1 The Survey of Enacted Curriculum and the Evaluative Study 
of NSF’s Urban Systemic Initiatives 

The Survey of Enacted Curriculum (SEC) is being conducted as a study component of a three year 
grant from the National Science Foundation (NSF), entitled “How Reform Works: An Evaluative 
Study of NSF’s Urban Systemic Initiatives (USI).” The purpose of the study is to determine the 
impact of the USI program on student achievement and the learning infrastructure in urban school 
districts. The study is focusing on development of an inferential causal model that relates the 
systemic initiative drivers and other key elements to the outcomes observed. 

As part of the study activities, the evaluative study team conducted four site visits each year to 
interview superintendents, principals, teachers, and USI and district staff members, and to observe 
classrooms and teacher practices. During two-day site visits to the Baltimore, Dallas, Detroit, 
and Phoenix school districts in 1999, and Columbus, Fresno, Memphis and Philadelphia in 2000, 
the study team met with a group of teachers to conduct the survey. This is the second Survey of 
Enacted Curriculum report summarizing all eight USI site survey data. 

Within the context of the USI evaluation study, there are three main objectives for conducting 
the Survey: 

• Collect data from a sample of teachers in selected urban schools to assess classroom 
practices and curriculum enactment in science and mathematics. This data will be used to 
confirm and validate changes in practice relevant to USI implementation. 

• Analyze enacted classroom curriculum and instruction in relation to expectations for practice 
as outlined in state/district content standards, science and mathematics reform initiatives, and 
standard-based assessment. 

• Analyze the effects of professional development on teaching practices within the context of 
USI implementation. 

The SEC is a practical research tool for collecting consistent data on mathematics and science 
instructional practices and curriculum based on teacher reports of what is taught in the classroom. 
The SEC offers an objective approach to analyzing current classroom practices in relation to state, 
national, and local content standards and the goals of systemic initiatives. The survey results 
provide a means of validating described changes in practice in urban systemic sites by analyzing 
classroom practice data from a sample of schools in selected sites. It also provides a method for 
analyzing the effects of professional development in actual classroom practice. 
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1.2 Background of Urban Systemic Initiatives and the Evaluative Study 

NSF launched the USI program in 1993, applying lessons learned from its initial State Systemic 
Initiative (SSI) program to the problems of inner city school systems. The USI program was 
offered to the 25 cities with the largest number of K-12 students living in poverty. Five cohorts of 
cities (22 cities in all) signed cooperative agreements with NSF for a five-year concerted system- 
wide effort to promote standards-based reform in mathematics, science, and technology (MST). 
The NSF investment was meant to be a catalyst for large-scale educational change affecting 
standards, curriculum, assessment, professional development, partnerships, and convergence of 
intellectual and fiscal resources, with the goal of improving student achievement. 

Over the course of its systemic initiative programs, NSF has developed a theoretical structure for 
systemic reform that is based on six “drivers”; four process drivers and two student outcome 
drivers, as well as a number of cross-cutting issues such as equity, quality, scaling-up, coordination 
and organization. NSF is committed to measuring impact within its systemic initiatives in the 
following three areas: 1) the implementation of standards-based curriculum, instruction and 
assessment, 2) enhanced professional development for teachers of mathematics and science and, 3) 
increased mathematics and science achievement of all students. 

With a focus on the six drivers outlined by NSF, our evaluative study is seeking evidence of 
program effectiveness that profoundly affects and sustains systemic reform in 22 USI cities. 

We are seeking to identify interrelationships among the four process drivers (Drivers 1 to 4) and 
two outcome drivers (Drivers 5 and 6) based on compiled outcome data collected by the Key 
Indicator Data System (KIDS). 



Urban Systennic Initiatives 




Cohort 93 (8) 
Cohort 94 (7) 
Cohort 95 (4) 
Cohort 97 (2) 
Cohort 99(1) 



Total (22 Sites) 
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Our study is focusing on two major hypotheses and one investigative question 

Hypothesis I: A well-implemented USI program has significant and sustainable positive impact 

on the infrastructure that supports student success (opportunity to learn) including 
curriculum, teacher skills, and resources in urban school districts. 

Hypothesis II: A well-implemented USI program leads to significantly improved student 
outcomes in MST. 

Question I: Which SI drivers or cross-cutting variables (equity, quality, scaling-up, etc.), 

are most critical to achieving sustainable reform and improved student outcomes, 
and how do these variables interact with each other? 



Six Drivers for NSF’s Educational System Reform 

1 . Implementation of comprehensive, standards-based curricula as represented in instructional 
practice, including student assessment, in every classroom, laboratory, and other learning 
experiences provided through the system and its partners. 

2. Development of a coherent, consistent set of policies, that supports: provision of high quality 
mathematics and science education for each student; excellent preparation, continuing education, 
and support for each mathematics and science teacher (including all elementary teachers); and 
administrative support for all persons who work to dramatically improve achievement among all 
students served by the system. 

3. Convergence of the usage of all resources that are designed for or that reasonably could be used 
to support science and mathematics education-fiscal, intellectual, material, curricular, and 
extra-curricular— into a focused and unitary program to constantly upgrade, renew, and improve 
the educational program in mathematics and science for all students. 

4. Broad-based support from parents, policymakers, institutions of higher education, business and 
industry, foundations, and other segments of the community for the goals and collective value of 
the program, based on rich presentations of the ideas behind the program, the evidence gathered 
about its successes and its failures, and critical discussions of its efforts. 

5. Accumulation of a broad and deep array of evidence that the program is enhancing student 
achievement, through a set of indices that might include achievement test scores, higher level 
courses passed. College admission rates, college majors, Advanced Placement Tests taken, 
portfolio assessment, and ratings from summer employers, and that demonstrate that students 
are generally achieving at a significantly higher level in science and mathematics. 

6. Improvement in the achievement of all students, including those historically underserved. 
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CHAPTER II 



METHODOLOGY AND INSTRUMENTATION 



11.1 Background 

Under the grant for the USI Evaluative Study, Systemic Research, Inc. contracted with the Council 
of Chief State School Officers (CCSSO) to adopt the Survey of Enacted Curriculum (SEC) as a 
study component for a three-year period beginning in Spring 1999. The SEC instruments, 
analysis, and reporting methods were designed and field-tested by CCSSO in collaboration with 
researchers at the National Institute for Science Education (NISE) at the University of Wisconsin- 
Madison. The design and development work began in 1995 as part of a multi-state collaborative 
project, and the current survey design was completed under support from the National Science 
Foundation, Education and Human Resources Directorate, Division of Research, Evaluation and 
Communication. 

11.2 Sampling and Data Collection 

The SEC collects teachers’ response to written survey forms. The survey forms were prepared in 
four categories: two subjects (mathematics and science) per two grade levels (elementary and 
middle). In each urban site, science and mathematics teachers were recruited from 20 schools -- 
10 elementary (grade 3, 4 or 5) and 10 middle (grade 6, 7 or 8). To form a control group for 
statistical analysis, a group of teachers were recruited from both high implementation schools 
(involved in USI implementation for a longer period of time with active participation in 
professional development) and low implementation schools (schools less actively involved in USI 
implementation) at each grade level. Two mathematics and two science teachers from each school 
were selected. The survey forms were distributed to 80 participating teachers: 20 mathematics 
and 20 science teachers per grade level. 

11.3 Survey Procedures 

The survey was conducted during site visits in 1999 (Baltimore, Dallas, Detroit, and Phoenix) and 
2000 (Columbus, Fresno, Memphis, and Philadelphia). Before each site visit. Systemic Research 
sent out an initial memorandum explaining the purpose and methodology of the survey. The USI 
Project Director in each site assisted in selecting the 20 schools, and recruiting the teachers to 
participate in the survey. In 1999, one teacher from each school was invited to attend a survey 
orientation session. After a full explanation of the survey, the teacher representatives were asked 
to return to their schools with four surveys (two each for mathematics and science) to distribute to 
three other teachers who were asked to complete it. The completed surveys were then returned by 
mail to the Wisconsin Center for Education Research for data processing and analysis. 
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Due to the low response (142 surveys returned from the 232 distributed), a different procedure 
was initiated in 2000. Instead of inviting one representative teacher from each school to explain 
the survey procedures and complete the survey at each school site, all of the teachers were invited 
to a central location to complete the survey. This resulted in a better response (248 surveys 
returned). Appendix C presents a summary of descriptive data on teachers participating in the 
survey during 1999 and 2000. 

A total of 390 teachers participated in the survey process; 142 in 1999 and 248 in 2000. 

An average of 49 teachers in each of the eight sites participated, with a wide range from 24 in 
Baltimore to 80 in Philadelphia. The demographic description of teachers is as follows: 

• 55% elementary school teachers, 45% middle school teachers 

• 49% mathematics teachers, 51% science teachers 

• 79% female mathematics teachers, 1 8% male mathematics teachers, 3% no response 

• 77 % female science teachers, 20% male science teachers, 3% no response 

• 46% underrepresented minority mathematics teachers 

• 36% underrepresented minority science teachers 



11.4 Survey Instrument Structure 

The survey was conducted using four different instruments to accommodate the two grade levels 
and two subject matters involved. However, each instrument consists of three sections as shown in 
Table 1. 



Table 1 . General Structure of SEC Instrument for Middle School Mathematics 



Section 


Questions (number of questions & type) 


I. 

Genera! School and 
Class Descriptions 


• School description (2 Q- multiple choice) 

• Class description (1 0 Q- multiple choice) 

• Most recent unit of mathematics instruction & instructional time 
distribution (1 0 Q- percentage of time) 


II. 

Subject Content - 
Expectations for Students 
in Mathematics 


• Number sense/ Properties/ Relationships (17 Q) 

• Measurement (13 Q) 

• Data analysis (15 Q) 

• Algebra concepts (22 Q) 

• Geometric concepts (1 8 Q) 

• Instructional technology (3 Q) 


III. 

Instructional Activities 
in Mathematics 


• Homework (1 0 Q) 

• % of mathematics instruction time (1 2 Q) 

• % of problem-solving activities (8 Q) 

• % of pairs or small groups work (6 Q) 

• % of use of hands-on materials (6 Q) 

• Use of calculators, computers, etc. (1 1 Q) 

• Assessments (8 Q) 

• Instructional influences (10 Q) 

• Classroom instructional preparation (1 8 Q) 

• Teacher opinions (1 8 Q) 

• Professional Development (1 4 Q) 

• Formal course preparation (3 Q) 

• Teacher characteristics (8 Q) 


Total 


• 242 Questions 
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Section I collects school and classroom information, along with the teacher’s most recent unit of 
mathematics instruction and instructional time distribution as shown in Figure 1 . 

Figure 1. Sample Questions in Section 



What percent of mathematics instructional time was spent on the following activities? 



Enter the percentage of time for each item in the box provided, so that items 9>16 total 100%. Then use 
the scale to code your response (rounded to the nearest 10%) for each item on the answer sheet. 





% 


None 


10 


20 


30 


40 


50 


60 


70 


80 


904 


9 Management or administrative routines, 
interruptions, and other non- instructional 




© 


© 


© 


0 


© 


© 


© 


© 


© 


© 


activities 
























10 Whole class lecture or class discussion 




© 


0 


© 


0 


© 


© 


0 


© 


© 


© 


'll Individual student work (e.g. completing 
exercises, reading textbook) 




© 


0 


© 


0 


© 


© 


© 


© 


© 


© 


12 Small group work 




© 


0 


© 


0 


© 


© 


© 


© 


© 


© 


13 Field study or out-of-class investigation. 




© 


0 


© 


0 


© 


© 


© 


© 


© 


© 


14 Student demonstrations or presentations. 




© 


0 


© 


0 


© 


© 


© 


© 


© 


© 


15 Review or work on homework during class. 




© 


0 


© 


0 


© 


© 


© 


© 


© 


© 


16 Test or quiz 




© 


0 


© 


0 


© 


© 


© 


© 


© 


© 



100% (Note: Total should sum to 1001 



Section II requests information regarding subject topic coverage and teacher’s expectations for 
students in the target class for the current year. It is not intended to reflect any recommended 
or prescribed content for the grade level. For middle school mathematics, six topic areas 
(ex. Number sense/properties/relationships) are detailed by multiple sub-topic areas (ex. Place 
value. Whole numbers. Operations, etc.) as shown in Figure 2. 



Figure 2. Sample Questions for Middle School Mathematics - 
Topic Coverage and Teacher’s Expectations for Students 
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Time on Topic j 


1 \Mid die School Afath T<^ics j 
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Expectationa for Snjdenta in Mathematics 


















<none> 






Number sense / Properties / Relationships Memorize 


Understand 

Concepts 


Perform 

Procedures 


Analyze } 
Rea.son 


Solve Novel 
Problems 


Integrate 




O 


0 


0 
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Place value 


0 


0 


0 


0 


0 


O 


0 


0 
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0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 
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0 


0 
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Whole numbers 
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To complete this section, the teacher identifies topic/sub-topic areas covered in his/her 
mathematics class, using “Time on Topic’’ column (0 = none- not covered, 1 = slight coverage- 
less than one class/lesson, 2 = moderate coverage- one to five classes/lessons, 3 = sustained 
coverage- more than five classes/lessons). Then the teacher indicates the relative emphases of . 
each expectation for students for every sub-topic taught using scale bubbles for six categories: 
memorize, understand concepts, perform procedures, analyze/reason, solve novel problems, and 
integrate. Appendix D presents detailed explanations of each student expectation category for 
mathematics and science. Four scale bubbles indicate: 0 = no emphasis, 1 = slight emphasis 
(accounts for less than 25% of the time), 2 = moderate emphasis (accounts for 25% to 33% of the 
time), 3 = sustained emphasis (accounts for more than 33% of the time). 

Section III asks ten types of questions regarding instructional practices. These topics include: 

• Homework 

• Instructional Activities in Mathematics 

• Use of Calculators, Computers, and Other Equipment 

• Assessments 

• Instructional Influences 

• Classroom Instructional Preparation 

• Teacher Opinions 

• Professional Development 

• Formal Course Preparation 

• Teacher Characteristics 

Figure 3 illustrates a sample question regarding “Instructional Activities in Mathematics.” 



Figure 3. Sample Questions about Instructional Activities in Mathematics 



INSTRUCTIONAL ACTIVITIES IN MATHEMATICS 










Listed below are some questions about what students in target class do in mathematics. For each activity. 




pick one of the choices (0, 1,2,3) to indicate the percentage of instructional time that students 


are engaged in 




the activity identified. Please think of an average student in this class. 


in responding. 




What percentage of mathematics instructional time in this target class do students: 






|NOTE; No more than two‘3‘s , or four ‘2‘e should be circled for any given si 


et of items. 






1 




None 


Less than 25% 


25% to 33% 


More than 33% 


^ Observe the teacher demonstrate how to do a 




o 
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procedure or solve a problem. 




^ Read about math-related topics in books, 
magazines, articles 
^ Collect and/or analyze data. 




o 




9 


30 Maintain a portfolio of their own work. 
Use hands-on materials/ mani pula tives (e.g. 




o 


9 
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counting blocks, geonxtric shapes, algebraic tiles, 
etc.). 




o 


9 
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^ Engage in mathematical problem solving (e.g. 










computation, story-problems, and mathematical 
investigations). 
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33 Stiulcnts take notes. 
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34 Work in small groups 




o 


9 


9 


35 Work on an assigned mathematics project at 
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home or away from school. 


W 




3® Use computers, calculators, and/or other 
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technology to learning mathematics. 
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37 Work individually on assignments. 
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38 Take a quiz or test 
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CHAPTER III 

SURVEY RESULTS 
SUMMARY AND ANALYSIS 



MI.1 Overview 

State and national standards in mathematics and science provided the foundation for the survey 
design, item writing, and reporting results. The standards used in design and reporting were 
selected by the CCSSO/NISE study team with a panel of mathematics and science education 
leaders from states. Not all topics and standards that provided the structure for the survey 
questions are reported here. The data analysis reported provides a snapshot of some of the 
practices and curriculum content in selected urban school districts. 

This report presents SEC results based on 191 participating teachers of mathematics and 199 
participating teachers of science in eight USI sites: Baltimore, Columbus, Dallas, Detroit, Fresno, 
Memphis, Philadelphia, and Phoenix. In all eight districts, the elementary school surveys were 
completed by teachers of grade 4 or 5, and middle school surveys by teachers of grade 7 or 8. 

One of the key elements of all of the USI programs across districts was professional development 
for teachers that focused on increasing teachers' content knowledge in science and mathematics 
and skills in pedagogy based on state and national standards. Recognizing the emphasis on 
professional development, our analysis of the Survey data compared teacher responses in two 
groups: a) those receiving High amounts of professional development (High PD) defined as 16 or 
more hours of professional development in their teaching subject in the last 12 months, and b) 
those receiving Low amount of professional development (Low PD) defined as less than 16 hours. 
These operational definitions are consistent with analysis of data from the NAEP 1996 survey of 
teachers (Shaughnessy, et al., 1997) and other national surveys such as the Schools and Staffing 
Survey. The cut point of 16 hours divided the sample of teachers fairly evenly into two strata. We 
also analyze the data by comparing teachers of middle grades and teachers of elernentary grades. 

III.2 Selected Item Profiles and Content Maps/Graphs for 
Mathematics and Science 

This section presents selected item profiles of SEC especially relevant to standards-based 
curriculum, instructions, and assessment. The theme or concept that is being reported and 
analyzed in each data chart is based on standards for learning in mathematics or science. The 
charts and data presentations are intended to illustrate some of the findings that were analyzed. 
The full survey results are attached as an appendix for further comprehensive analysis. Table 2 
outlines the 18 data charts presented in this section which focus on: 
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• Active Learning in Science 

• Reasoning and Problem Solving in Mathematics 

• Science and Mathematics Content Maps and Graphs 

• Multiple Assessment Strategies in both Science and Mathematics 

• Use of Lab and Technology 

• Influences on Curriculum and Practices 

• Participation in Professional Development 

• Teacher Preparation 

Item Profile Legend 

Except the science and mathematics content maps and graphs, item profiles summarize the 
average of each item response using a bar graph referencing a group mean and standard 
deviations. Each chart summarizes the item response using a three-column presentation. The first 
column presents item response summaries by grade level; elementary and middle school. The 
second and third columns present item response summaries of elementary and middle schools by 
High PD and Low PD teachers. Each cell presents two bars comparing two groups of data. 

The vertical line in the center indicates the item (group) mean and the shaded area (blue or gray) 
shows the range of responses from -1 to +1 standard deviation (StD) from the mean, which 
generally includes two-thirds of the survey responses. A bordered bar indicates a statistically 
significant mean difference between the two groups. 

Profile Legend Bordered Bar 

Group 

-1 StD Mean +1 StD r — - . ^ — i 

I I I i'll 



Summary Scales 

Several charts present Summary Scales that present an average of combined item responses 
relevant to standards-based curriculum and instruction. The summary scale has been statistically 
transformed so that the grand mean of all teachers has a value of zero. Two bars present relative 
group means and standard deviation compared to the grand mean. 

Summary Scales 

I— I +1 StD 



0 «Grand Mean» 

— Group Mean 

—I -1 StD 

Scale Reliability: .80 
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Scale Reliability 

Scale reliability is a summary measure of internal reliability among the selected profile items. 
For example, if one item is higher than average, the scale reliability predicts how likely the other 
items in the scale will also be higher than average. Reliability ranges from 0 to 1 . As a rule of 
thumb, anything in the .8 to .9 range is considered high reliability. 



Table 2. Selected Item Profiles and Content Maps/Graphics 



Chart 


Title 


Chart 1 


Active Learning in Science 


Chart 2 


Reasoning and Problem Solving in Mathematics 


Chart 3 


Science Content Maps 


. Chart 4 


Elementary School Science Content Graphs 


Chart 5 


Middle School Science Content Graphs 


Chart 6 


Mathematics Content Maps 


Chart 7 


Elementary Mathematics Content Graphs 


Chart 8 


Middle School Mathematics Content Graphs 


Chart 9 


Multiple Assessment Strategies in Science 


Chart 1 0 


Multiple Assessment Strategies in Mathematics 


Chart 1 1 


Use of Lab Equipment & Educational Technology in Science 


Chart 12 


Use of Educational Technology in Mathematics 


Chart 13 


Influences on Curriculum and Practices in Science 


Chart 14 


Influences on Curriculum and Practices in Mathematics 


Chart 1 5 


Professional Development in Science 


Chart 1 6 


Teacher Preparation in Science 


Chart 17 


Professional Development in Mathematics 


Chart 1 8 


Teacher Preparation in Mathematics 
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Chart 1: Active Learning in Science 

Chart 1 displays a summary of item profiles to the survey question ''When students in the target 
class are engaged in laboratory activities, investigations, or experiments as part of science 
instruction, what percentage of that lab time do students The Item Profiles display the 
percentage of time spent on classroom activities relevant to active learning in science. Data 
related to Active Learning in Science are reported in two ways: the eight individual item profiles, 
and a Summary Scale that combines the four item profiles (as indicated by *) relevant to 
standards-based curriculum and instruction. 

Active learning is one of the central concepts in state and national standards for student learning in 
science that underlies the Survey. The data in Chart 1 illustrate the extent of active science 
learning and the kinds of classroom practices being used as reported by science teachers. 
Differences in active learning strategies employed by teachers across grade levels and High PD 
vs. Low PD are displayed. 

• Summary Scale: The summary scale in active learning in science is comprised of the 
following four standards-based item profiles: use science equipment or measuring tools, 
change something in an experiment to see what will happen, design ways to solve a problem, 
and make prediction, guesses or hypotheses. 

The results of the survey indicate that there is no difference in the amount of time spent on 
the above referenced items by any group of teachers. Middle school teachers with High PD 
report slightly more, although not statistically significant, emphasis on standards-based 
instruction. 

• Experiments/Investigations: The Item Profiles display the percent of time classes spend 
on specific activities during active learning in science, particularly during experiments or 
investigations. Another set of questions in the survey asked about how class time is typically 
allocated (see Appendix Chart B.7. Use of Class Time During Most Recent Unit of 
Instruction in Science.) In total, classes used 27% of science class time doing hands-on or 
laboratory activities in science. In a typical classroom experiment or investigation, students 
would be expected to be engaged in a number of these activities in combination or sequence. 
Thus, it is useful to examine groups of activities with similar time frames. In Chart 1, we 
see that in general the four items associated with standards-based instruction are practiced 
nearly 50% of the time during active learning. No one activity dominated time spent- all 
activities were reported between almost 10% to 15% of time. In our survey the amount of 
professional development a teacher received did not have an impact on the time spent on any 
particular activity. 

Summary: The data on active learning in science show that all teachers are utilizing standards- 
based instructional methods nearly half of the time. There is not much variation in time spent on 
any of the eight items listed. 
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Chart 1 

Active Learning in Science 



Profile Legend 
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Group 

Mean 
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Eight District USI Sample--(Cohorts 1 & 2) 


By Grade Level 


Elementary Sch. 


1 Middle Sch. 


Middle (93) 

I I Elementary (106) 


EZ] High PD (45) 

1 □ Low PD (61) 


□ High PD (56) 

1 — 1 Low PD (37) 



Summary Scale 



0 



4-1 StD 



« Grand Mean » 
Group Mean 

1 StD 

i Scale Reliability: .68 

Item Profile 



m 



When students are engaged in laboratory activities, investigations, or experiments, what portion of 
that time are students engaged in the following? 



Follow step-by-step directions.’ 



*Use science equipment or 
measuring tools. 



Collect data. 



^Change something In an experiment 
to see what will happen. 



^Design ways to solve a problem.- 



*Make predictions, guesses, or 
hypotheses. ' 



Make tables, graphs, or charts. -- 



Draw conclusions from science data. 




0% 10% 20% 30% 0% 10% 20% 30% 0% 10% 20% 30% 



' item included in Summary Scale 
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Chart 2; Reasoning and Problem Solving in Mathematics 

Both reasoning and problem solving are curriculum standards for K-12 mathematics education set 
out in the NCTM Mathematics Standards (1989), and this learning goal is found in most state 
content standards for mathematics (Blank, et al., 1997). Key questions concerning classroom 
practices and instruction related to the standard are: '*How much instructional time do teachers 
spend on this goal ? '' 'What do teachers mean when they use the term 'problem solving'? " 

The results in Chart 2 illustrate how results from the Survey that can answer these questions. 

• Summary Scale: The scale on reasoning and problem solving is comprised of four survey 
items that, taken together, provide a reliable index of the degree to which teachers are 
focusing mathematics instruction on improving students’ reasoning and problem solving 
skills and knowledge using standards-based methods. These four items are solve novel 
mathematical problems^ apply mathematical concepts to real or simulated "real-world" 
problems, make estimates, predictions, guesses, or hypothesis or analyze data to make 
inferences or draw conclusions. The scale results show that middle school classes 
experience more problem solving activities than elementary school classes. The data also 
show that mathematics classes of elementary school High PD teachers spend more time on 
mathematics reasoning and problem solving than the classes of Low PD teachers. The 
results of middle school teachers show the opposite effect; that Low PD teachers spend more 
time on reasoning and problem solving skills than High PD teachers. However, this data is 
not statistically significant. 

• Problem-solving Activities: When analyzing the results from questions on mathematical 
reasoning and problem solving, it is important to know how teachers interpret the term 
"problem solving.” For example, do they mean completing exercises in a text or solving 
novel problems (those for which students have no ready procedure)? The item profile 
section of the chart identifies these distinctions. For example: about 15% of time in 
elementary and middle grades mathematics in problem solving activities is spent on both 
computational exercises from a text or worksheet and solve word problems from a textbook 
or worksheet. There is wide variation in time on these activities, from less than 10% to over 
25%, as indicated by the range of responses for one standard deviation from the mean. In 
general the mathematics classes of High PD teachers spend less time on these activities 
compared to Low PD teachers. 

All teachers report the majority of time spent on problem solving and reasoning skills 
involve standards-based methods. In general, middle school teachers spend significantly 
more time than elementary school teachers on problem solving, but interestingly, PD has a 
more marked effect on elementary school teachers; those with High PD spend much more 
time on standards-based problem solving than their Low PD counterparts. 

Summary: The findings on Mathematics Problem solving and Reasoning indicate that teachers 
with more professional development in mathematics content and teaching report a greater 
instructional emphasis on standards-based instruction, such as applying mathematical concepts to 
real world problems and making predictions and less on drill and practice exercises. 
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Chart 2 

Reasoning and Problem Solving in Mathematics 



Profile Legend 
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Eight District USI Sample - (Cohorts 1 & 2) 


By Grade Level 


Elementary Sch. 


Middle Sch. 
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Summary Scale 
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0 « Grand Mean » 
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Scale Reliability; .68 



Item Profile 



I When students are engaged in problem-solving activities, what portion of that time are students 
I engaged in the following? 



j Complete computational exercises or procedures .. 

from a text or a worksheet. 



Solve word problems from a 
textbook or worksheet. 



•Solve novel mathematical problems. .. 



Write an explanation to a problem 
using several sentences. 



•Apply mathematical concepts to real 
or simulated "real-world” problems. 



•Make estimates, predictions, 
guesses, or hypotheses. 



•Analyze data to make inferences 

or draw conclusions. ' 
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■* Item included in Summary Scale 
Bordered bar indicates statistically significant mean difference. 
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Charts 3, 4, & 5: Science Content Maps and Content Graphs 

The Survey of Enacted Curriculum incorporates an innovative approach to data collection on 
subject content taught in class. Teachers are asked to report the amount of time spent on content 
in mathematics or science using a “subject content matrix.” The two dimensions of the matrix are 
content topics and expectations for students. Teachers first identify the major mathematics or 
science standards covered in class and amount of time spent teaching specific topics, and then 
report the expectations they had for student learning (e.g., memorize, understand concepts, 
perform procedures, analyze information). Teachers are asked to identify subject content taught in 
one class over one school year. (See Appendix A for a detailed explanation of content matrix.) 

Data on the mathematics and science content taught in class have been compiled and visualized 
using “content maps,” and “content graphs.” A content map, as shown in Figure 4, provides a two- 
dimensional representation of instructional content using a surface area chart, which results in a 
graphic very similar to topographical maps. The grid overlaying each map identifies a list of 
topics areas (indicated by horizontal grid lines- O) and six categories of cognitive expectations for 
students (indicated by vertical grid lines- ©). The intersection of each topic area and category of 
cognitive expectation represents a measurement node. A Content Map is an excellent instrument 
for reporting a general overview using a 3-dimensional perspective. 



Figure 4. A Sample Mathematics Content Map 




D 



Expectation for Students 



For more detailed node-by-node comparison, a Content Graph is presented in a matrix format 
combining multiple bar graphs on each node with scales (see sample Chart 4). Each bar graph 
compares High PD and Low PD teachers. Each row total shows a mean value and a range of one 
standard deviation for each topic area. Likewise, each Column Total shows a mean and standard 
deviation for six categories of cognitive expectations for students. 

Content maps and content graphics are both generated from teacher responses across all eight USI 
school sites. The teacher responses are aggregated and reported by grade level (elementary and 
middle) and High PD vs. Low PD teachers 

The main advantages of content matrix design for data collection on content taught is its 
consistency with standards-based learning, as described in national professional standards — 
National Council of Teachers of Mathematics (NCTM), 1989; American Association for the 
Advancement of Science (AAAS), 1993; National Research Council (NRC), 1995 - and most 
state standards. With this approach, curriculum is viewed as a combination of the mathematics or 
science disciplinary knowledge to be learned (e.g., geometry) and the skills and capacities that 
students are expected to gain through instruction (e.g., solve real-world problems). 

Chart 3 presents four science content maps for both middle and elementary science teachers by 
High PD and Low PD. Charts 4 and 5 present content graphs comparing High PD and Low PD in 
bar graph format. 

• Elementary School Science. The areas of Life Science were reported by teachers as being 
taught from 20 to 22% of the time, with High PD teachers spending slightly less time. 
Physical Science and Earth Science were reported at about 14 to 19% of time. Measurement 
and Calculation in science averaged 14% of time. The aggregate category Nature of Science 
averaged nearly 20% of time. This category includes teaching scientific method, history of 
science, science and technology, and science-health-environment. The most striking aspect 
of the data on science topics is the wide variation in time on topics spent by teachers, 
particularly in Nature of Science with time spent varying from 0 to over 30% of time. 

All six expectations for science learning were reported as over 10% of time; understand 
concepts, conduct experiments, analyze information and apply concepts were all reported as 
20% or more of time. Memorize was reported as the least expected kind of learning, which 
includes learning facts, definitions, terms, and formulas. High PD teachers reported slightly 
more time on expectations for analyze information, but other expectations were very similar 
between the two groups. 

• Middle School Science. Teachers reported spending the most amount of time on Life Science 
(27%) followed closely by Physical Science (22%). Less than 5% of time was spent on 
Measurement and Calculation in Science. There were wide variations among teachers in amount 
of time spent on most topics, particularly the 10% to 40% time variation on Life Science. 

Teachers’ expectations for student learning were fairly balanced among the six expectations. 
More time was spent on understand concepts and the least amount of time on the traditional 
practice of memorize. 
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Summary: Elementary teachers who participated in the survey reported fairly balanced content 
among the areas of Life^ Physical^ and Earth Science, and almost the same amount of time in 
instruction on Nature of Science. Only small differences were found between High PD and Low 
PD teachers. There was very little difference among Middle school High PD and Low PD 
teachers in both content and expectations. 



Chart 3 



Science Content Map 
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Chart 4 



Elementary School Science Content Graph 
Eight District Sample (Cohort 1 & 2): Low PD (36), High PD (32) 
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Chart 5 



Middle School Science Content Graph 
Eight District Sample (Cohort 1 & 2): Low PD (27), High PD (54) 




I I Low PD teachers 
I I High PD teachers 



Marginals Legend 




Marginals = The sum across rows or columns. 





Charts 6, 7 & 8: Mathematics Content Maps, and Graphs 

Chart 6 presents four mathematics content maps for middle school by High PD and Low PD. 
Charts 7 and 8 present content graphs for the same data. In our survey the majority (82%) 
of middle level mathematics teachers were teaching grade 8. The content maps reflect patterns 
of responses from teachers with High PD in mathematics in the past year vs. Low PD in 
mathematics. In the content graphs (Charts 7 & 8), each cell has bars representing average 
percent time for High PD vs. Low PD teachers, and the row and column marginals indicate the 
mean and range of responses for each topic area (e.g., number sense) and type of expectation 
(e.g., memorize). 

Understand Concepts refers to students’ ability to represent a concept, apply it to a problem, or 
explain its use. Perform procedures means using numbers for counting or ordering, doing 
computation, or solving equations. Solve novel problems indicates doing non-routine problems or 
those for which the student has no routine strategy or algorithm. 

• Elementary School Mathematics: Teachers reported spending about 20% of time on 
number sense, operations and geometric concepts. They spent about 18% of time on 
measurement. Teacher expectations for students were highest for understand concepts, and 
perform procedures and lowest for solve novel problems. 

• Middle School Mathematics: Teachers reported spending the most instructional time on 
number sense (average 20 to 24% time), geometric concepts (22%), and data analysis 
(20%). Algebraic concepts were reported at about 18% of time. Further analysis of the 
content matrix data (not shown) indicate that the specific topics reported most often under 
number sense were fractions, decimals, percent, and ratio and proportions, and topics 
covered most often in Algebra were use of variables and multi-step equations. 

Teaching of Algebra and Geometry is highly varied among teachers/classes, especially for High 
PD teachers. High PD teachers spent slightly more time teaching Algebra on average. Time 
reported on teaching Algebraic Concepts varied from less than 8% to more than 30% of time 
across classes of High PD teachers. Time on Geometry varied from 12 to over 30% of time in 
both groups of classes. High PD teachers had more variation in time spent on Number Sense. 

Expectations for learning as reported by teachers focused most on understand concepts and 
perform procedures (about 20%). Teachers with High PD had large variations in expectation 
for memorizing facts and terms in mathematics class, but surprisingly, in general reported 
higher expectations than Low PD teachers. 

Summary: There were no significant differences between High PD and Low PD teachers. All 
teachers reported spending the most amount of time on number sense, operations and geometric 
concepts. Most emphasis was put on understand concepts and perform procedures expectations. 
Elementary school teachers reported the most amount of time spent on the content areas of 
number sense, operations, and geometric concepts. Middle school teachers reported spending 
class time on all five major content areas with the least amount on measurement. Differences 
among middle High PD and Low PD teachers were mainly in the greater degree of variation in 
teacher responses among High PD teachers in four of the five content areas. The average amount 
of time spent teaching each topic area are similar. 
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Chart 6 

Mathematics Content Maps 
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Chart 7 



Elementary School Mathematics Content Graph 
Eight District Sample (Cohort 1 & 2): Low PD (53), High PD (46) 
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Chart 8 

Middle School Mathematics Content Graph 
Eight District Sample (Cohort 1 & 2): Low PD (22), High PD (52) 
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Charts 9 & 10: Multiple Assessment Strategies in Science and Mathematics 

In standards-based reform initiatives, including USI, mathematics and science teachers are 
encouraged and prepared to make use of a variety of assessment strategies rather than relying on a 
single type of assessment, such as paper-and-pencil tests comprised of objective items or routine 
procedural problems. The purpose of varied assessment strategies is, in part, to increase the 
validity of the inferences that teachers can make about student learning. Using multiple sources of 
evidence allows the strengths in one type of assessment to compensate for weaknesses in another. 
But to what extent are mathematics and science teachers moving beyond a reliance on a single 
type of assessment, and what strategies are they using? The results depicted in Charts 9 and 10 
illustrate how SEC resources provide data on these questions. 

Science: The Summary Scale on Multiple Assessment Strategies in Science (Chart 9) is based on 
several Survey items: extended response items for which student must explain or justify answers, 
performance task or events, individual or group demonstration or presentation, science projects or 
portfolios. Together these items provide a reliable index of the degree to which science teachers 
are using less traditional multiple assessment strategies in the classroom. The results show wide 
variation in use of multiple assessment strategies. However, at both elementary and middle levels, 
the teachers with High PD report greater use of multiple assessments than their counterparts with 
Low levels of PD. There is a statistically significant difference among High PD and Low PD 
elementary school teachers. 

• Science Item Profiles: The responses from elementary school science teachers on specific 
items concerning assessment strategies show distinct differences between High PD and Low 
PD teachers in the following four areas: extended response, performance tasks, portfolios, 
and systemic observation of students. High PD teachers are more likely to use these 
strategies than Low PD teachers. Only one of the strategies, systemic observation, is 
considered to be a traditional method of assessment. We also note wide variation in the use 
of these strategies; from never, up to three times a week. 

Middle school teachers also report wide variations in the use of multiple assessments, but no 
real difference was reported between High PD and Low PD teachers. The teachers in our 
survey use the traditional practice of systemic observation of students most often. Use of 
this method, along with portfolios, show the widest variation of use. 

Mathematics: The Summary Scale on Multiple Assessment Strategies in Mathematics (Chart 10) is 
based on several Survey items: extended response items for which students must explain or justify 
answer, performance task or events, individual or group demonstration or presentation, and mathe- 
matics projects. Together these items provide a reliable index of the degree to which mathematics 
teachers are using multiple assessment strategies in the classroom. The results show that middle 
school teachers are much more likely to employ nontraditional assessment strategies. There is a slight, 
not sigmficant, more likelihood of High PD teachers than Low PD teachers to use these methods. 

• Mathematics Item Profiles: The responses from mathematics teachers on several of the 
assessment questions are displayed in the second section of the chart. For each type of 
assessment strategy there is wide variation in use among teachers from never to several times 
per week. It is also notable that the average teacher uses several methods of assessing 
student knowledge and skills in mathematics. 
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Elementary school teachers use performance tasks or events more than middle school teachers but 
there is no significant difference between High PD and Low PD teachers. While middle school 
teachers are less likely to use these methods of assessment, high PD teachers use performance 
tasks much more often than Low PD teachers (an average of 1 - 4 times per year vs, more than 3 
times a month.) Other notable difference is the use of multiple assessments between High PD and 
Low PD middle school teachers are in the areas of extended response, mathematics projects. 
High PD teachers are more likely to use these strategies. 

All teachers report using multiple assessment strategies, with wide variations of use among all 
groups. The traditional practice of systemic observation is the method utilized most often; 
somewhat surprising is the relatively small level of use of multiple choice items (average 1- 4 
times per year to 1 - 3 times per month.) 
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Chart 1 0 

Multiple Assessment Strategies in Mathematics 
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Charts 11 & 12: Use of Lab Equipment & Educational Technology in Science 
and Mathematics 

An important indicator of active, inquiry-based methods of teaching science and mathematics, 
as well as school system capacity for supporting this approach, is the availability and use of 
educational technology and laboratory equipment. Science and mathematics standards advocate 
learning to apply knowledge to real problems and to gain skills that will be used outside of school. 
Science and mathematics applications in careers now involve computers, calculators, and a variety 
of simple and complex lab equipment. Thus, a key component of the Survey of Enacted 
Curriculum concerns the use of equipment and technology in teaching science and mathematics. 
Charts 1 1 and 12 illustrate how the survey data can be analyzed to examine several kinds of 
questions concerning availability and instructional uses of educational technology and lab 
equipment. 

Science: Two items on teacher reports of use of science equipment in classrooms illustrate 
diiferences highlighted in Chart 1 1 : 

• Use of Equipment: Middle school teachers are significantly more likely to use lab 
equipment, but there is wide variation— from never to weekly. High PD teachers report 
slightly more use. Running water (a traditional indicator of lab capacity) is not available at 
all in about one-third of elementary classes (the bar shows middle two-thirds of responses), 
and in the average class running water is rarely used. The average middle grade science 
class uses running water less than monthly, and rates vary from rarely to weekly. 

Recently, “high-tech” approaches to experimentation in classrooms involve computer-lab 
interfacing devices, often called computer-based labs (CBLs). The sample data in the study 
indicate that one-third of classes did not have CBLs available, while the average class had 
access but rarely used them. Use at the middle grades level is highly varied — from not 
available to monthly or greater rates of use. 

• Educational Technology in Science: Chart 1 1 illustrates the extent of calculator, 
computer, and other educational technology use in science classes and how they are applied 
in instruction. The scale shows wide variation among science classrooms. The only 
significant difference between elementary and middle school teachers is that middle school 
teachers require students to display and analyze data more often than elementary school 
students. 

In science classes, calculators and computers are most often used to learn facts or practice 
procedures and displaying and analyze data. However, these uses vary widely, from 0% to 
42% of time, showing that teachers make very different use of educational technology. 

Mathematics: The Summary Scale in Chart 12 includes three items that ask teachers to report the 
extent of calculator and computer use in class and how they are applied in instruction: use sensors 
or probes, collect data or information, and display and analyze data. The scale shows wide 
variation among mathematics classrooms. The amount off* professional development a teacher 
received had a significant effect at the middle school level- High PD teachers use more 
technology. 
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• Calculator use: Teacher reports on student use of calculators indicate that middle grades 
consistently use them at least monthly, while elementary classes less than seven times per 
year. Graphing calculators are used much less often than traditional calculators. Few 
elementary schools have them available, and middle school use varies widely from not 
available to monthly. 

• Item Profiles: The Survey data show that educational technology's most frequent use in 

mathematics instruction learning facts or practicing procedures (average 25%). Other 

frequent uses are display and analyze data, take a quiz or test, and use individualized 
instruction or tutorial software. Less than one third of instructional time is spent on the 
standards-based methods of use sensors or probes, collect data or information, and display 
and analyze data. 
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Chart 11 

Use of Lab Equipment & Educational Technology in Science 
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Chart 1 2 

Use of Lab Educational Technology in Mathematics 
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Charts 13 & 14: Influences on Curriculum and Practices in Science 
and Mathematics 

The Survey includes questions for teachers aimed at the main influences on their classroom 
curriculum. In some schools, textbooks and materials that are selected by districts, states, 
or schools may be a major influence on what is taught. Many states and districts have established 
standards for student learning in mathematics and science, and these standards have been guided 
or influenced by national standards developed and published by professional organizations. Even 
where widely disseminated and used by teachers, standards do not provide curricula for teaching 
in classrooms. Teachers may rely on district curriculum, their own knowledge and experience, 
their colleagues, or mandated assessment programs to determine what is taught. The question of 
influences on curriculum is important for analyzing commonalities in curriculum and for 
determining how change and improvements can be made in science and mathematics education. 

Charts 13 and 14 provide Cross-Site results for science and mathematics reported by teachers in 
the eight USI sites concerning major influences on their curriculum and teaching practices. 

• Influences on Science: The summary scale shows that state and national standards have 
more of an influence on High PD teachers than Low PD teachers especially in elementary 
schools. 

The item profiles for science teachers indicate that state and district frameworks or 
standards had the greatest positive influence, as well as national standards and preservice 
preparation for science. Textbooks/instructional materials and state tests had little influence 
on science curriculum reported by teachers. District tests were reported as having a negative 
influence. At both levels, PD made little difference in most responses concerning curriculum 
influences. 

• Influences on Mathematics: Summary scale results for state and national standards' 
influences on mathematics indicate little difference among all groups of teachers. 

The item profiles indicate that the most consistent positive influences reported by teachers 
were state and district frameworks or standards. District tests had the least, but unlike in 
science, a positive influence on curriculum. Differences between High and Low PD teachers 
were not notable. 
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Chart 1 3 

Influences on Curriculum and Practices in Science 
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Chart 1 4 

Influences on Curriculum and Practices in Mathematics 
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Charts 15, 16, 17, & 18: Professional Development and Teacher Preparation 



In every USI site, professional development has been recognized as a major driver of school 
reform reaching to the classroom level and directly impacting student outcomes. The prime 
purpose of professional development is to improve teachers’ content knowledge, pedagogy skills, 
including inquiry-based teaching, and methods of assessing student learning. According to the 
Urban School Key Indicators of Science and Mathematics Education (September 2000, Systemic 
Research, Inc.), 75% (118,399) of teachers working in USI schools attended some form of 
professional development during the 1998-99 school year. Middle school mathematics and 
science teachers had the highest rates of participation in professional development activities as 
shown in Figure 5. A major portion of USI funds (average 48%) was spent on Professional 
Development activities across all of the USI sites. 

Figure 5 
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A key section of the Survey of Enacted Curriculum requests individual teacher’s information 
regarding professional development activities, such as the amount of professional development 
activities (clock hours) by categories (content, curriculum materials, assessment, etc.) in the past 
year, and the impact on their actual teaching practices. The questions also ask about their formal 
course preparation in terms of the number of quarters or semesters of subject specific courses 
taken at the undergraduate and graduate level. 

Another section of the survey. Classroom Instructional Preparation, asks for teacher views on how 
well prepared they consider themselves for using a variety of teaching strategies and for teaching 
with various groups of students. The cross-site survey results for science teachers are shown in 
Charts 15 and 16, and for mathematics teachers in Charts 17 and 18. 
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Chart 15: Professional Development in Science 

• Professional Development Participation: As shown in the first section of Chart 15, 
middle school science teachers received close to 16-35 hours of professional development 
both in in-depth study of science content and methods of teaching science. Middle school 
science teachers’ PD participation was significantly greater than elementary teachers who 
averaged six hours less. 

• Professional Development Categories: As shown in Chart 15, 80 to 90% of High PD 
teachers participated in professional development on content standards, in-depth study of 
science content, implementing curriculum, multiple strategies for assessment, new methods 
of teaching, and educational technology. About 60% of Low PD teachers attended these 
types of activities. In addition, over 60% of High PD teachers attended an extended institute 
of 40 hours or more and teacher network or study group. 

• Impact on Instruction: Teachers reported that professional development activities had 
impact on their teaching practices. The majority of elementary and middle school science 
teacher reported the impact of PD activities they participated in as between trying to use and 
changed teaching practice, especially among High PD teachers. In general, those teachers 
who participated in an extended institute reported the higher rate of changes in their teaching 
practices. 

Chart 16: Teacher Preparation in Science 

• Teacher Major and Preservice Course Taking: The average elementary teacher in the 
survey had less than one courses in biology/life science, physics/chemistry/physical science, 
and geology /astronomy /earth sciences, and three to four science education courses. 

The average middle school teacher had one to four science content courses and seven to 
eight science education courses. Almost 40% of middle school science teachers graduated 
with a degree in science or science education compared to less than 5% of elementary 
teachers. In all instances. High PD teachers reported having more science courses than Low 
PD teachers suggesting their higher level of interest in science. 

• Science Teaching Preparedness: Teachers reported on their preparedness to teach science 
in various ways. For most of the teaching strategies, teachers responded from somewhat 
prepared to very well prepared. Middle school teachers are more prepared than elementary 
school teachers to use cooperative learning. Overall, High PD teachers feel much more 
confident in their preparedness compared to Low PD teachers especially in provide science 
instruction that meets science standards, manage a class of students using hands-on or 
laboratory equipment, and use a variety of assessment strategies. All three items show 
statistically significant mean differences between High PD and Low PD teachers as indicated 
by the bordered bars. 
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Chart 1 5 

Professional Development in Science 
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Teacher preparation in Science 
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Chart 17: Professional Development in Mathematics 



• Professional Development Participation: Middle school mathematics teachers received an 
average of 16-35 hours of professional development on in-depth study of math content and 
methods of teaching math during the last twelve months. Elementary teachers averaged less 
than 6-15 hours in each area. 

• Professional Development Categories: As shown in Chart 17, 80 to 90% of High PD 
teachers participated in professional development on content standards, implementation of 
new curriculums, new methods of teaching, in-depth study of math content, meeting the 
needs of students, and multiple strategies for assessment. This participation pattern is similar 
to those of science teachers, except for slightly lower participation rates in educational 
technology. About 60% of Low PD teachers attended these type of activities. About 55 to 
60% of High PD teachers reported participating in an extended institute-style PD of 40 hours 
or more and teacher network or study group. 

• Impact on Instruction: Mathematics teachers reported that professional development 
activities had impact on their teaching practices. Almost all of the High PD teachers 
reported that PD activities resulted in them trying to use or changed teaching practice. The 
impact of professional development on instruction was clearly stronger among the High PD 
teachers than their Low PD peers. 

Chart 18: Teacher Preparation in Mathematics 

• Teacher Major Preservice Course Taking: As shown in Chart 18, approximately 20% of 
middle school mathematics teachers reported having a major in mathematics or mathematics 
education compared with less than 1% of elementary school teachers. High PD teachers in 
middle school have significantly more intensive backgrounds in mathematics than low PD 
teachers, which may indicate a voluntary selection of PD among better prepared teachers of 
mathematics. Both High PD and Low PD middle school teachers reported taking three to 
four refresher math courses, one to two advanced math courses, and three to six math 
education courses at the undergraduate and graduate level. In elementary school. Low PD 
teachers took slightly less numbers of courses compared to High PD teachers. Overall, 
middle school math teachers took more mathematics courses as undergraduate and graduate 
students. 

• Mathematics Teaching Preparedness: Teachers reported that they were well prepared to 
teach mathematics using various strategies. There were significant dilferences between High 
PD and Low PD middle school teachers in six categories including use cooperative learning, 
integrate mathematics with other subjects, use a variety of assessment strategies, teach 
estimation strategies, teach problem solving strategies, select instructional materials, and 
teach with manipulative materials. Clearly, intensive professional development has a 
positive impact on teachers’ preparedness. 
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Chart 17 

Professional Development in Mathematics 
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Chart 18 

Teacher Preparation in Mathematics 
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III.3 Other Item Profiles 

The remaining part of the SEC data not covered in the previous section has been compiled in 
28 charts attached in Appendix B as follows: 



Chart B.1 : 


Scale Measures of Instructional Practice - Science 


Chart B.2: 


Scale Measures of Instructional Practice - Mathematics 


Chart B.3: 


Scale Measures of Teacher and School Characteristics - Science 


Chart B.4: 


Scale Measures of Teacher and School Characteristics - 
Mathematics 


Chart B.5: 


Class Description - Science 


Chart B.6: 


Class Description - Mathematics 


Chart B.7: 


Use of Class Time During Most Recent Unit of Instruction 
in Science 


Chart B.8: 


Use of Class Time During Most Recent Unit of Instruction 
in Mathematics 


Chart B.9: 


Use of Homework in Science 


Chart B.10: 


Use of Homework in Mathematics 


Chart B.11 : 


Instructional Activities in Science 


Chart B.1 2: 


Instructional Activities in Mathematics 


Chart B.1 3: 


Small Group Work in Science 


Chart B.14; 


Small Group Work in Mathematics 


Chart B.1 5: 


Student Reflection on Scientific Ideas 


Chart B.1 6: 


Use of Hands-on Materials in Mathematics 


Chart B.1 7: 


Teacher Readiness - Science 


Chart B.1 8: 


Teacher Readiness - Mathematics 


Chart B.1 9: 


Teacher Opinions - Science 

a) Beliefs About Student Learning, b) Professional Collegiality 


Chart B.20: 


Teacher Opinions - Mathematics 

a) Beliefs About Student Learning, b) Professional Oollegiality 
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Chart B.21 : 


Grade 4 Science - Cohort 1 Content Maps 


Chart B.22: 


Grade 4 Science - Cohort 2 Content Maps 


Chart B.23: 


Grade 4 Mathematics - Cohort 1 Content Maps 


Chart B.24: 


Grade 4 Mathematics - Cohort 2 Content Maps 


Chart B.25; 


Grade 8 Science - Cohort 1 Content Maps 


Chart B.26: 


Grade 8 Science - Cohort 2 Content Maps 


Chart B.27: 


Grade 8 Mathematics - Cohort 1 Content Maps 


Chart B.28: 


Grade 8 Mathematics - Cohort 2 Content Maps 
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CHAPTER IV 
CONCLUSIONS 



IV.1 Summary of Findings 

The findings of the Year 2000 Survey of Enacted Curriculum analysis can be summarized in three 
areas; teaching practices, curriculum and subject content, and teacher professional development 
and preparation. 



Teaching Practices 

The data on Active Learning in Science show that all teachers are utilizing standards-based 
instructional methods nearly half of the time. Survey results indicated that the amount of professional 
development a teacher received did not have an impact on the time spent on any particular activity. 

The findings on Mathematics Problem Solving and Reasoning indicate that teachers with more 
professional development in mathematics content and pedagogy report greater instructional time 
on standards-based instruction, such as applying mathematical concepts to real world problems 
and making predictions and less time on drill and practice exercises. In general, middle school 
teachers spend significantly more time than elementary school teachers, but professional 
development has a more marked effect on elementary school teachers; those with High PD spend 
much more time on standards-based problem solving than their Low PD counterparts. 

Scale results show wide variation in use of Multiple Assessment Strategies in science and 
mathematics. In science at both the elementary and middle levels the teachers with High PD 
report greater use of multiple assessments than their counterparts with Low levels of PD, 
especially at the elementary level. For mathematics, the results show that middle school teachers 
are much more likely to employ nontraditional assessment strategies. High PD teachers are more 
likely than Low PD teachers to use these methods. 



Curriculum and Subject Content 

For science, state and district frameworks or standards have the greatest positive influence on 
curriculum, as well as national standards and preservice preparation. Teachers reported little 
influence of textbooks/materials, professional development^ and state tests. District tests were 
reported as having a negative influence. 

In mathematics, the most consistent positive influences reported by teachers were state and district 
frameworks or standards. District tests had the least, but unlike in science, a positive influence on 
curriculum. Differences between High and Low PD teachers were not notable. 
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Teacher Professional Development and Preparation 

Science: Survey results indicate that professional development received by teachers, and 
emphasized by USI, is largely being used and applied in classrooms. The middle school teachers 
who participated in this survey reported an average amount of 16-35 hours of professional 
development while elementary school teachers reported an average of 6-15 hours. Over 80% of 
High PD teachers participated in professional development on content standards, implementing 
curriculum, multiple strategies for assessment, new methods of teaching, and educational 
technology. About 60% of Low PD teachers reported receiving professional development in 
similar areas. The professional development activities were reported by teachers as having an 
impact on their teaching practices. The majority of teachers indicated that they were trying to use 
the new knowledge or it had changed their practice. 

In general, middle school teachers were much more likely to have had a science major in 
preservice education than elementary school teachers- 40% versus less than 5%. Also, middle 
school teachers reported taking an average of eight to 12 science and science education courses 
compared to the average three to four similar courses for elementary school teachers. 

Most of the teachers responded that they felt somewhat to very well prepared to teach science. 
Overall, High PD teachers felt more confident in their preparedness than Low PD teachers. 

Mathematics: Mathematics teachers’ professional development participation rates are the same as 
those in science- ranges of 16-35 hours for middle school and 6-15 hours for elementary school 
teachers. More than 80% of High PD teachers received professional development in content 
standards, implementation of new curriculum, new methods of teaching, in-depth study of math 
content, meeting the needs of students, and multiple strategies for assessment. About 60% of Low 
PD teachers attended similar types of activities. Mathematics teachers reported that professional 
development had an impact on their classroom practices. Almost all of the High PD teachers 
reported that the activities resulted in them trying to use new strategies or they had changed their 
teaching practices. 

Approximately 20% of middle school mathematics teachers reported having a major in 
mathematics or mathematics education compared with less than 1% of elementary school teachers. 
Middle school teachers averaged seven to 12 mathematics or mathematics education courses in 
college. Elementary school teachers took slightly less number of similar mathematics courses. 

Teachers reported being well prepared to teach mathematics at both the elementary and middle 
school levels. High PD teachers were significantly more likely to report that they were well 
prepared in several categories especially those related to inquiry-based learning. 

IV.2 Uses and Misuses of Data 

Data on enacted curriculum cannot itself provide a vision of quality education, but it can inform 
that vision by providing indicator measures to serve as guideposts and measuring sticks to 
determine where things are and where they need to go in order to move closer to the goals 
selected. By providing a broad selection of measures pertaining to content, pedagogy, climate, 
and background, survey data allows concerned individuals and groups to construct their own 
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individualized set of indicators for determining how local practice compares to their and others’ 
visions of quality practice. 

The survey data also allows schools to determine estimates of time spent engaged in various types 
of activities, and descriptive information about how these activities are provided. Together this data 
forms both an operational definition of desired practice and a description of current practice. 

While data from these surveys provide a wealth of valuable information about what goes on in 
classrooms, care must be taken not to over-interpret or even misinterpret the information provided 
from the surveys. 



Curricuium Content Data 

The content matrix portion of the Survey of Enacted Curriculum represents the most thorough and 
detailed approach to measuring classroom subject content that has been developed and tested 
using a teacher self-report methodology. The method is the first to effectively incorporate both 
content topics and expectations for learning in teacher reporting on what is taught in class. This 
matrix approach is critical to analyzing mathematics and science curriculum and teaching in order 
to relate classroom content to state and national standards developed in the 1990s. Leading 
educators have long known that curriculum is not simply a listing of topics but rather includes 
what students are expected to learn about the topics and what skills and knowledge they must 
master. 

There are issues in obtaining valid, reliable data using the content matrix method. Teachers have 
to report their curriculum in terms of a common set of categories and time scales. The survey 
requires concentration and time to complete. As the surveys have been field tested, the design has 
been changed and simplified. The analysis of data from the USI surveys in 1999 and other teacher 
surveys recently conducted indicate that the content matrix methodology does work. We have 
found that a comprehensive picture of classroom curriculum across schools and districts can be 
obtained using this approach. 



Specific Classes 

The survey data represents teacher perceptions of classroom practice targeted to a particular class 
and group of students. It should be kept in mind when interpreting such data that teacher reports 
for a particular class may not represent that teacher’s practice across all classes and students. 



Socially Desired Responses 

In considering the limitations of the survey data, it should be noted that the accuracy of the data 
depends upon the accuracy of teachers’ perceptions and estimates. Accuracy can also be affected 
by perceptions of socially desirable responses, particularly if sanctions or rewards are 
commensurate with certain practices. 



Misuse of Data 

The primary misuse of data on the enacted curriculum or teaching practices is the use of such data 



for accountability purposes. To be a useful tool for school improvement, this type of indicator 
data should stand aside from the accountability mechanisms developed by schools, districts, or 
states. In large part, this is due to the possibility that the diagnostic potential of such data would 
be compromised if used for accountability purposes because of the increased likelihood of 
teachers providing socially desirable responses to the survey questions. Further, the indicators that 
such measures provide should not be rigidly imposed as a definition of quality practice. Survey 
data on curriculum must be combined with local contextual information in order to provide a 
sufficient basis for making decisions about changing, rewarding, or sanctioning individual 
practice. For these reasons this indicator data is best conceived of as a diagnostic and not an 
accountability tool. 
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APPENDIX A: 



INTERPRETING CONTENT MAPS 



Content maps provide a three-dimensional representation of instructional content using a surface 
area chart which results in a graphic very similar to topographical maps. The grid overlaying 
each map identifies a list of topics areas (indicated by horizontal grid lines; see O in Exhibit I) 
and six categories of cognitive expectations for students (indicated by vertical lines; see O). The 
intersection of each topic area and category of cognitive expectation represents a measurement 
node (see ©). Each measurement node indicates a measure of instructional time for a given topic 
area and category of cognitive expectation based upon teacher reports. The resulting map is based 
upon the values at each of these measurement nodes. It should be noted that the spaces between 
each measurement node, that is the surface of the map, are abstractions and are not based upon 
real data, the image of the map is simply a computer generated graphic based upon the values for 
each intersecting measurement node. The map display is utilized to portray the third dimension 
(percent of instructional time; see ©) onto this grid utilizing shading and contour lines to indicate 
the percent of instructional time spent (on average across teachers) for each topic by cognitive 
expectation intersection. 

The increase (or decrease) in instructional time represented by each shaded band is referred to as 
the measurement interval (see ©). To determine the amount of instructional time for a given 
measurement node, count the number of contour lines between the nearest border and the node, 
and multiply by the measurement interval. 

The graphic at left below displays the three dimensional counterpart of the image represented by 
the content map displayed on the right. Both graphs indicate that Understanding Concepts related 
to Number Sense and Operations occupies the majority of time spent on grade four mathematics 
instruction (9% or more of instructional time over the course of a school year). 
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Exhibit I 
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Chart B.1 

Scale Measures of Instructional Practice - Science 
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Chart B.2 

Scale Measures of Instructional Practice - Mathematics 
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Chart B.3 

Scale Measures of Teacher and School Characteristics - Science 
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Chart B.4 

Scale Measures of Teacher and School Characteristics - Mathematics 
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Chart B.5 

Class Description - Science 
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Chart B.6 

Class Description - Mathematics 
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Elementary Math 
(Percent of teachers reporting) »> 




0 20 40 60 80 100 




0 20 40 60 80 100 




How many students are in this class? 



Response Codes: 



1 Hi ‘’'*1';. ’■ I 



— r — I — r — r~ 

1 2 3 4 5 






~I 1 ) f 

2 3 4 5 






“1 F i r r 

1 2 3 4 5 



0 = 10 or less 
3 = 21-25 



1= 11 - 15 
4 = 26-30 



2= 16-20 
5 = 31 or more 



During a typical week, 
approximately how many hours will 
this class spend in mathematics 
instruction? 

(Number of Instructional hours) »> 



I r r I r r r 

01 2345678 









7 — I — — r”r~T — f — r~r“r~r~i~r~r“i 

012345678 012345678 



What is the average length of each 
class period for this class? 



Response Codes: 



1 



r I [ r f 

2 3 4 5 6 






r r r [ r f 

0 1 2 3 4 5 6 






r r I \ \ 

2 3 4 5 6 



0 = N/A 1=30-40min. 2 = 41-50min. 

3 = 51 -60 min. 4 = 61-90 min. 5 = 91 - 120 min. 

6 = Varies due to block scheduling or integrated instruction. 



Bordered bar Indicates statistically significant mean difference. 
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Chart B.7 

Use of Class Time During Most Recent Unit of Instruction in Science 



Leqend Group 

Mean 



I « n 

-1 StD +1 StD 



Eiqht District USi Samole-fCohorts 1 & 2) 


By Grade Level | Elementary Sch. 


Mid^e^h. j 


r~l Middle (93) I □ High PD (45) 

I I Elementary (106); □ Low PD (61) | 


□ High PD (56) 
1 1 Low PD (37) 



I How many class periods did 
I your most recent science 
I instructional unit cover? 



Response Codes: 



: 


: ■ 




i ! 1- 

1 r r f ( 


■1 ! I- 

r 1 1 r 1 


I 1 • 

■■ "f f f f 1 



012345 012345 012345 



! 0=1-2 


1=3-5 


2=6-10 


\ 3=11-15 


4 = 16-20 


5 = 21 or more 



What percent of science instructional time in this unit was spent on the following? 



Administrative routines, 
interruptions, and other non- ^ 
instructional activities. 

Whole class lecture or class.... 

discussion. 

Individual student work (e.g. 
completing exercises, reading 
textbook). 

Non-laboratory small group work. ' 



Student work with hands-on, . 
manipulative, or laboratory materials. 

Field study or out-of-class ... 
investigation. 




Review or work on homework during 
class. 



Test or quiz. 






Bring students up to date due to... 
absences or transfers. 



\ f r 

0% 20% 40% 












3 



3 



~i \ r r f 

0% 20% 40% 












r i i 

0% 20% 40% 



Bordered bar Indicates statistically significant mean difference. 
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Chart B.8 

Use of Class Time During Most Recent Unit of Instruction in Mathematics 



Legend Group 
Mean 



-1 StD +1 StD 



1 Eight District USI Sample - (Cohorts 1 & 2) 


By Grade Level | 


Elementary Sch. 


Middle Sch. 


i □ Middle (84) I 

i IGI Elementary (107) i 


□ High PD (36) i 

□ low PD (71) 


I I High PD (56) 

I I Low PD (28) 



How many class periods did your 
most recent mathematics 
instructional unit cover? 








F r * ; 


GG ; 


h r-i : 








( 


f r f r [■■■■( 

) 1 2 3 4 5 ( 


f f f f f 

) 1 2 3 4 5 ( 


I 1 1 1 1 

) 1 2 3 4 5 


Response Codes: 


0= 1-2 
3=11-15 


1=3-5 2 = 

4 = 16-20 5 = 


6-10 i 

21 or more | 



I What percent of mathematics instructional time in this unit was spent on the following? 



Administrative routines, 
interruptions, and other" 
non-instructional activities. 

Whole class lecture or class 
discussion. 

Individual student work (e.g. 
completing exercises, reading 
textbook. 



Small group work. 



Field study or out-of-class 
investigation" 

Student demonstrations or 
presentations." 

Review or work on homework 
during class. 



Test or quiz. 



Bring students up to date due to 
absences or transfers. 






; 



dZ]; 

Gz 

G3 



"f r f r r 

0% 20% 40% 



B; ; 



S : 
S; 
Zb i 



“f i r f f 

0% 20% 40% 












1 — f — r — r — \ 
0% 20% 40% 



Bordered bar Indicates statistically significant mean difference. 





Chart B.9 

Use of Homework in Science 



Legend Group 

Mean 



-1 StD 



+1 StD 



Eiqht District USi Samole-iCohorts 1 & 2 ) 


By Grade Level 


1 Elementary Sch. 


Middle Sch. 


ED Middle (93) 

I I Elementary (106) 


1 ED HighPD (45) 
□ Low PD (61) 


□ High PD (56) 
j □ Low PD (37) 



How many minutes does the 
typical student spend on a 
normal homework assignment? 



Response Codes: 



n — r — r — f 

2 3 4 5 



S' 



r r f f f 

1 2 3 4 5 



\ 0 = No homework assigned 1-< 15 minutes 
i 3 = 31-60 minutes 4 = 61-90 minutes 



E3 



“i r~f I f 

0 1 2 3 4 5 

2 = 15-30 minutes 
5 = > 90 minutes 



How often do you usually assign 
science homework? 






ZF 



Response Codes: 



r r f f i [ i f r r r ( 

01234 01234 01234 

[ 6 = Never 1= < Once per week 2 = 1-2 times/week 



Does homework count towards 
student grades? 



ciE 



Response Codes: 



0 1 2 3 0 1 

1 0 = Never 1= Usually does not 



2 3 0 

2 = Usually does 



_r r f 

1 2 3 

3 = Always does 



L 



What percentage of the time that students spend on science homework do they: 



Read about science in books, 
magazines, or articles. 



Answer questions from a science book or 
worksheet. 



Solve science problems that involve 
computation. 

Revise and improve students' own work. 
Collect data or information about science. 

Write about science. 












S3 






I i~ 






0% 20% 40% 0% 20% 40% 0% 20% 40% ; 



Bordered bar Indicates statistically significant mean difference. 
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Chart B.10 

Use of Homework in Mathematics 



Legend 



Group 

Mean 



-1 StD 



+1 StD 



Eight District USI Sample - (Cohorts 1 & 2). 


I 1 

By Grade Level 


Elementary Sch. 


Middle Sch. 


j o Middle (84) 

i □ Elementary (107) I 

1 ] 


□ High PD (36) 

□ Low PD (71) 


1 1 High PD (56) 

1 1 Low PD (28) 



How many minutes does the 
typical student spend on a 
normal homework assignment? 



Response Codes: 



0 1 



~i r r [ 

2 3 4 5 






0 1 



1 r f 

2 3 4 5 






_I . ^ . j 

0 1 2 3 4 5 



0 = No homework assigned 1- <15 minutes 
3 = 31-60 minutes 4 = 61-90 minutes 



2= 15- 30 minutes 
5 = > 90 minutes 



How often do you usually assign 
mathematics homework? 






: ^ 


: 


: : I y a 


: : : : i 

j 






(iff 

) 1 2 3 4 1 


rill 

D 1 2 3 4 C 


I f [ r 1 

) 1 2 3 4 1 


Response Codes: 


0 = Never 1= < Once per week 2 = 1-2 times/week | 

I 3 = 3-4 times/week 4 = Every day \ | 


Does homework count towards 
student grades? 

( 

Response Codes: 








i — 




! 








r r r 

D 1 2 3 ( 


r 1 1 

D 1 2 3 ( 


i 1 o 1 

) 1 2 3 1 


0 = Never 1= Usually does not 2 = Usually does 3 = Always does \ j 



I What percentage of the time that students spend on mathematics homework do they 

icziii 



Do arithmetic computation or procedures. 



Show steps required in doing a procedure or 
solving an equation. 



Explain their reasoning or thinking in solving a 

problem. 



Work on a demonstration, presentation, or 
proof of their mathematics work. 



Collect data or information as part of 
mathematics homework. 



Write a report for a mathematics project. 




r r~ 

0% 20% 40% 









CO 






dO 



EZ? 









UJ 



CP 






I 1 r 1 1 f r r 

0% 20% 40% 0% 20% 407o 



Bordered bar Indicates statistically significant mean difference. 
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Chart B.11 

Instructional Activities in Science 



•-egend 


Eight.District USI Sample-(Cohorts 1 & 2) | 


Mean 

1 7Z 1 


By Grade Level 


Elementary Sch. | Middle Sch. 


1 a 1 

-1 StD +1 StD ! 

! 

I 


ffl Middle (93) 

i [ Elementary (106) 


□ High PD (45) | □ High PD (56) 

□ Low PD (61) 1 QLowPD (37) 



What percent of science instructional time do students in the class engage in the 
following activities? 



Listen to the teacher explain 
something about science. 



Read about science in books, 
magazines, or articles. 



Collect or analyze data. 



Maintain and reflect on a science . 
portfolio of their own work. 



Write about science. • 



Do a laboratory activity, investigation,. 

or experiment in class. 



Watch the teacher give a 
demonstration of an experiment. 



Work in pairs or small groups 
(non- laboratory). * 



Do a science activity with the class 
outside the classroom or laboratory. 



Use computers, calculators, or other 
technology to learn science. 



Work individually on assignments. ' 



Take a quiz or test. 



m 






JHL 



\ i 

0% 10% 20% 









Ed? 









Ed 






I I 

0 % 10 % 20 % 



E?b 



a 



Ed? 
d 



Z? 



? 



dd 



EE 






0 % 10 % 20 % 



Bordered bar Indicates statistically significant mean difference. 
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Chart B.12 

Instructional Activities in Mathematics 



Legend 



Group 

Mean 



-1 StD 



+1 StD 



Eight District USI Sample - (Cohorts 1 & 2) i 


By Grade Level 


Elementary Sch. 


Middle Sch. 


rn Middle (84) | 

I } Elementary (107) 


1 □ High PD (36) 

: □Low PD (71) 


[ — 1 High PD (56) | 

[ — 1 Low PD (28) j 



What percent of mathematics instructional time do students in the class engage in the 
following activities? 



Watch the teacher demonstrate how 
to do a procedure or solve a problem. 



Read about mathematics in books, magazines, 

or articles. ' 



Collect or analyze data. 



Maintain and reflect on a mathematics 
portfolio of their own work. 



Use hands-on materials or manipulatives (e.g. 
counting blocks, geometric shapes, algebraic 

tiles). 

Engage in mathematical problem 
solving (e.g. computation, story-problems, 
mathematical investigations. 



Take notes. 



Work in pairs or small groups. 



Do a mathematics activity with the class outside 
the classroom. 



Use computers, calculators, or other technology 
to learn mathematics. 



Work individually on assignments. 



Take a quiz or test. 









j--* 















f f 

0 % 10 % 20 % 









u 












z:? 



EO 

ZZI2: 






0 % 10 % 20 % 









Hi 




m; 




□ 




[EZE 

1 ; 


i 


1 1 



0 % 10 % 20 % 



Bordered bar Indicates statistically significant mean difference. 
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Chart B.13 

Small Group Work in Science 



Legend 

Group 

Mean 


Eight District USI Sample--(Cohorts 1 & 2) 


By Grade Level 


Elementary Sch. 


Middle Sch. 


I 1 1 

-1 StD +1 StD 


m Middle (93) 

1 1 Elementary (106) 


□ High PD (45) 
I — I Low PD (61) 


i □ High PD (56) 

1 1 Low PD (37) 



I When students work in pairs or small groups as part of science instruction, what percentage of 
I that time do students: 



I Talk about ways to solve science problems. 



Complete written assignments from the.. 

textbook or worksheets. 



Write results or conclusions of a.. 

laboratory activity. 

Work on an assignment, report, or project 
that takes longer than one week to ' 
complete. 

Work on a writing project where group 
members help to improve each other’s (or‘ 
the group’s) work. 

I Review assignments, problems, or prepare. 

for a test or quiz. 










E3 



ri-J-,- -L- 

r* - r 

si^zb 



0% 



20% 40% 0% 



20% 40% 0% 



20% 40% 



Bordered bar Indicates statistically significant mean difference. 




64 




Chart B.14 

Small Group Work in Mathematics 



‘-egend 

Mean 



-1 StD +1 StD 



Eight District USI Sample - (Cohorts 1 & 2) 



By Grade Level 



Elementary Sch. 



Middle Sch. 



C3 Middle (84) 

□ Elementary (107) 



□ High PD (36) 

□ Low PD (71) 



□ High PD (56) 

□ Low PD (28) 



When students in the target class work in pairs or small groups as part of mathematics 
instruction, what percentage of that time do students: 



Tak about ways to solve mathematics 
problems 







Complete written assignments from the 
textbook or worksheets. 



Work on an assignment, report, or project 
that takes longer than one week to oomplete. 



Work on a writing project where group 
members help to improve each others’ (or the 
group’s) work. 

Review assignments, problems, or 
prepare for a test or quiz. 



0% 20% 40% 



r^-r-q — I 

ES 

0% 20% 40% 



EE3 



0% 20% 40% 



Bordered bar Indicates statistically significant mean difference. 
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Chart B.15 

Student Reflection on Scientific Ideas 



Legend 



Group 

Mean 



-1 StD 



a I 

+1 StD 



Eight District USI Sample --(Cohorts 1 & 2) 


By Grade Level 


Elementary Sch. 


Middle Sch. 


Q Middle (93) 

I I Elementary (106) 


□ High PD (45) 
I I Low PD (61) 


□ HighPD (56) 

1 1 Low PD (37) 



When students collect informationaboui science from books, magazines, computers, or other sources, 
what percentage of that time do students; 



Ask questions to improve understanding. - 



Organize and display the information in 
tables or graphs. 



Make a prediction based upon the . 

information or data. 



Discuss different conclusions from the ■ 
information or data. 

List positive (pro) and negative (con) 
reactions to information. 



Reach conclusions or decisions based 
upon the information or data. 



0% 20% ' 40% 0% 20% ' 40% 0% 20% 40% 
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Chart B.16 

Use of Hands-on Materials in Mathematics 



Legend 



Group 

Mean 



-1 StD 



+1 StD 



Eight District USj Sample, -.(Co 



By Grade Level 


Elementary Sch. 


Middle Sch. 


H Middle (84) 

\ I Elementary (107) 


i □ High PD (36) 

i □ Low PD (71) 


I I High PD (56) 

I I Low PD (28) 



When students are engaged in activities that involve the use of hands-on materials, 
what percentage of that time do students: 



Work with hands-on materials such as 
counting blocks, geometric shapes, or 
algebraic tiles to understand concepts. 



Measure objects using tools such as rulers 
scales, or protractors. 



Build models or charts. 



Collect data by counting, observing, or. 

conducting surveys. 



Present information to students concerning a 
mathematical idea or project. ' 



0% 



20% 40% 













0% 20% 40% 



0% 20% 40% 



Bordered bar Indicates statistically significant mean difference. 
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Chart B.17 

Teacher Readiness - Science 



Legend 



Group 

Mean 



-1 StD +1 StD 



Eight District USI Sample--(Cohorts 1 & 2) 


By Grade Level i Elementary Sch. 


Middle Sch. 


n Middle (93) i □ High PD (45) 

1 1 Elementary (106)| Low PD (61) 


□ High PD (56) I 

I I Low PD (37) I 



For the following items, please indicate how well prepared you are now to: 



Teach students with physical disabilities. 



Teach classes for students with diverse 
abilities. - 



Teach science to students from a 
variety of cultural backgrounds. 



Teach science to students who have 
limited English proficiency. 



Teach students who have a learning 
disability which impacts science learning. 



Encourage participation of females in 
science. 



Encourage participation of minorities in- 
science. 



0 1 






1 



H — — 












0 1 



0 1 






I 



Response Codes; 0 = Not well prepared 1=Somewhat prepared 2= Well prepared 
I 3 = Very well prepared 



Bordered bar Indicates statistically significant mean difference. 
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Chart B.18 

Teacher Readiness - Mathematics 



Legend 



Group 

Mean 



-1 StD +1 StD 



Eiaht District USI Sample - (Cohorts 1 & 2). . . 


By Grade Level Elementary Sch. 


Middle Sch. 


r~l Middle (84) □ High PD (36) 

I — I Elementary (107) | □ Low PD (71) 


□ High PD (56) ■ 

I I Low PD (28) 



For the following items, please indicate how well prepared you are now to: 



Teach students with physical disabilities. 



Teach classes for students with diverse 
abilities. 



Teach mathematics to students from a 
variety of cultural backgrounds. 



Teach mathematics to students who have 
limited English proficiency. 



Teach students who have a learning disability 
which impacts mathematics learning. 



Encourage participation of females in - 
mathematics- 



Encourage participation of minorities in 
mathematics. 









s 






rjiiiziij-:' 









cS 



[£EZ 









cS 



nr 



; ^£7 



— [ [ f 

0 1 2 3 0 1 2 3 



Response Codes: 



0 = Not well prepared 1=Somewhat prepared 2= Well prepared 
3 = Very well prepared 



Bordered bar Indicates statistically significant mean difference. 
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Chart B.19 



Teacher Opinions - Science 

a) Beliefs About Student Learning, b) Professional Collegiality 



Legend 



-1 StD 



Group 
Mean 

+1 StD 



Plst 


rict USI Sample --(Cohorts 1 & 2) 


By Grade Level 


Elementary Sch. 


Middle Sch. 


I I Middle (93) I 

Elementary (106) i 


□ HighPD (45) 

I I Low PD (61) 


n High PD (56) 

□ Low PD (37) 




Bordered bar Indicates statistically significant mean difference. 
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Chart B.20 



Teacher Opinions - Mathematics 
a) Beliefs About Student Learning, b) Professional Collegiality 



Legend 



Group 

Mean 



-1 StD 



+1 StD 



Eight District USI Sample - (Cohorts 1 & 2) 


By Grade Level 


Elementary Sch. 


Middle Sch. 


m Middle (84) 

I I Elementary (107) 


I I High PD (36) 

i i Low PD (71) 


1 □ High PD (56) 

1 □ Low PD (28) 



Please indicate your opinion about each of the statements below: 



Students learn mathematics best when 
they ask a lot of questions. 



Students master and retain mathematical 
algorithms more efficiently through 
repeated practice than through the 
use of applications and simulations. 

Calculator use should be incorporated 
only after the mastery of basic 
arithmetic facts. 



All students can learn challenging 
mathematics content. 



Students learn mathematics best in 
classes with students of similar abilities. 



It is important for students to 
learn basic mathematics skills before 
solving problems. 



I am supported by colleagues to try 
out new ideas in teaching mathematics. 



Mathematics teachers in this school 
regularly share ideas and materials. 



Mathematics teachers in this school 
regularly observe each other 
teaching classes. 

Most mathematics teachers in this 
school contribute actively to making 
decisions about the mathematics 
curriculum. 

I have adequate time during the regular 
school week to work with my peers on 
mathematics curriculum or instruction. 













~r 



0 12 3 4 



0 = Strongly disagree 
3 = Agree 



Response Codes: 

Bordered bar Indicates statistically significant mean difference. 
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1 = Disagree 
4 = Strongly agree 



2 = Neutral / Undecided 
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Chart B.21 

Grade 4 Science - Cohort 1 Content Maps 



Phoenix Public Schools (6)* 






Nature of Science 

Measurement & 
Calculation in Science 

Life Science 
Physical Science 
Earth Science 




Percent of Instruction/! est 





Teacher Expectations 




Baltimore School District (5)* 




Detroit School District (12)* 




Teacher Expectations 



* (n) = Sample Size 



72 
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Chart B.22 

Grade 4 Science - Cohort 2 Content Maps 



Columbus Public Schools (13)* 





Fresno Unified School District (18)* 





Teacher Expectations 



* (n) = Sample Size 
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Chart B.23 

Grade 4 Mathematics - Cohort 1 Content Maps 



Phoenix Public Schools (1 1 )* 





Baltimore School District (5)* 





Teacher Expectations 



* (n) = Sample Size 
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Chart B.24 

Grade 4 Mathematics - Cohort 2 Content Maps 



Columbus Public Schools (15)* 




Fresno Unified School District (16)* 




Memphis City Schools (15)* 




Number Sense / 
Properties / Relationships 

Operations 
Measurement 
Algebraic Concepts 

Geometric Concepts 

Data Analysis, 
Probability, Statistics 

Instructional Technology 




Percent of Instruction/Test 




Measurement Interval = 1% 




Teacher Expectations 



School District of Philadelphia (19)* 




Teacher Expectations 



* (n) = Sample Size 




75 



84 



Chart B.25 

Grade 8 Science - Cohort 1 Content Maps 




Phoenix Public Schools (6)* 

Nature of Science 

Measurement & 

Calculation in Science 

Life Science 

Physical Science 
Earth Science 

Chemistry 

Dallas Schools (9)* 





Percent of Instruction/Test 





Teacher Expectations 



er|c 



Baltimore School District (7)* 




Detroit School District (9)* 




Teacher Expectations 



* (n) = Sample Size 
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Chart B.26 

Grade 8 Science - Cohort 2 Content Maps 



Columbus Public Schools (6)* 





Percent of Instruction/Test 





Teacher Expectations 



Fresno Unified School District (16)* 





Teacher Expectations 



* (n) = Sample Size 
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Chart B.27 

Grade 8 Mathematics - Cohort 1 Content Maps 



Number Sense / Properties / 
Relationships 




Measurement 

Data Analysis, Probability, 
Statistics 

Algebraic Concepts 
Geometric Concepts 



Instructional Technology 




Baltimore School District (7)* 




Number Sense / Properties / 
Relationships 







Measurement 

Data Analysis, Probability, 
Statistics 

Algebraic Concepts 



Geometric Concepts 



[nstruclional Technology 



Dallas Schools (12)* 





Measurement Interval = 1% 




Teacher Expectations 




Teacher Expectations 



* (n) = Sample Size 
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Chart B.28 

Grade 8 Mathematics - Cohort 2 Content Maps 



Number Sense / Properties / 
Relationships 

Measurement 

Data Analysis, Probability, 
Statistics 

Algebraic Concepts 
Geometric Concepts 
Instructional Technology 




Columbus Public Schools (2)* 





Fresno Unified School District (13)* 




School District of Philadelphia (20)* 




Teacher Expectations 



* (n) = Sample Size 
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APPENDIX C 



DESCRIPTIVE DATA ON TEACHERS PARTICIPATING 
IN THE SURVEY OF ENACTED CURRICULUM 



Survey of USI Sites 


1999 


2000 


TOTAL 




Baltimore, Dallas, 


Columbus, Fresno, 


8 USI Sites 




Detroit, & Phoenix 


Memphis, & Phila. 



Teachers Responding to Survey (whole 
or in part) 


1999 


2000 


TOTAL 


Number of Teachers 


Math 


Science 


Math 


Science 


Math 


Science 


Elementary 


37 


38 


70 


68 


107 


106 


I Middle 


32 


35 


52 


58 


84 


93 


Total 


69 


73 


122 


126 


191 


199 


I Grand Total 


142 


248 


390 



Teacher Demographics 


1999 


2000 


TOTAL 




Math % 


Science % 


Math % 


Science % 


Math % 


Science % 


Female 


76.8 


72.6 


80.3 


80.2 


79.1 


77.4 


1 Male 


18.8 


21.9 


18.0 


18.3 


18.3 


19.6 


White 


39.1 


56.2 


51.6 


62.7 


45.4 


60.3 


i Minority 


47.7 


38.3 


44.3 


34.9 


46.0 


35.7 



Class Reported by Teacher (1 class perteacher) 

1999 


2000 


TOTAL 


Number of Teachers 


Math 


Science 


Math 


Science 


Math 


Science 


Grade 2 


3 


2 


0 


3 


3 


5 


|3 


3 


7 


7 


9 


10 


16 


4 


26 


6 


56 


51 


82 


57 


|5or6 


7 


25 


7 


7 


14 


32 


7 


13 


6 


6 


8 


19 


14 


1 8 or higher 


16 


24 


45 


47 


61 


71 



Teaching Time 






1999 




2000 


TOTAL 






Math % 


Science % 


Math % 


Science % 


Math % 


Science % 


Elementary (hours/week) 


Less than 4 


4.3 


30.1 


4.9 


27.8 


4.6 


28.7 




|4-4.9 


2.9 


9.6 


5.7 


20.6 


4.7 


16.6 




5 or more 


43.4 


11.0 


46.7 


5.6 


45.5 


7.5 


Middle (hours/week) 


Less than 4 


4.3 


5.5 


3.3 


3.2 


3.6 


4.0 




|4-4.9 


15.9 


15.1 


6.6 


12.7 


9.9 


13.6 




5 or more 


26.0 


27.4 


31.9 


27.8 


32.8 


27.6 
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Teacher Characteristics 




1999 




2000 


TOTAL 






Math % 


Science % 


Math % 


Science % 


Math % 


Science % 


Experience:Years jn Subject 

Oto 2 


20.2 


26.0 


15.5 


13.5 


17.3 


18.1 




3to5 


14.5 


17.8 


24.6 


19.8 


20.9 


19.1 




6 to 11 


14.5 


24.6 


24.6 


33.3 


21.0 


30.2 




1 1 2 or more 


46.4 


27.4 


34.4 


31.7 


38.8 


30.2 


Highest Degree 


BA/BS 


49.3 


42.5 


54.9 


48.4 


52.9 


46.2 




1 MA/MS or higher 


43.4 


53.5 


44.2 


49.2 


44.0 


50.7 


Major: Bachelors or Highest 


Elementary Ed 


44.9 


27.4 


59.0 


41.2 


54.0 


36.2 




Middle Ed 


4.3 


4.1 


5.8 


1.6 


4.3 


2.5 




Math Ed 


13.0 


2.5 


6.3 




Science Ed 


12.3 


6.3 


8.5 




Mathematics 


4.3 


5.0 


4.7 




Science Field 


16.4 


8.8 


11.5 


Math Ed & Mathematics 


5.7 


1.6 


3.1 


Science Ed & Science Field 


5.5 


4.8 


5.0 


Other 


20.2 


28.8 


23.8 


32.6 


22.5 


32.1 



Teacher Professional Development 




1999 




2000 


TOTAL 




Math % 


Science % 


Math % 


Science % 


Math % 


Science % 


Content study in field 

(hours in last year) Less than 6 


33;3 


37.0 


29.5 


38.1 


30.9 


37.7 


|6 - 15 hours 


27.5 


30.1 


23,0 


22.2 


24.6 


25.1 


16 or more 


34.8 


30.1 


47.5 


38.9 


42.9 


35.7 


Methods of teaching in field 

(hours in last year) Less than 6 


27.5 


42.5 


27.1 


37.3 


27.3 


39.2 


|6 - 15 hours 


29.0 


24.7 


25.4 


25.4 


26.7 


25.1 


16 or more 


37.7 


28.7 


47.6 


35.7 


43.9 


33.2 
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APPENDIX D 



EXPECTATIONS FOR STUDENTS IN MATHEMATICS AND SCIENCE 

1. Mathematics 



Memorize 


• Facts 

• Definitions, Terms 

• Formulas, procedures 


Understand 

Concepts 


• Explain, define or represent concepts 

• Apply concepts in procedures and problems 

• Explain procedures, algorithms, solutions, strategies 

• Develop/explain relationships between concepts 

• Show or explain relationships between models, diagrams, 
or other representations 


Perform 

Procedures 


• Use numbers to count, order, denote 

• Do computational procedures or algorithms 

• Follow procedures/instructions 

• Solve equations/formulas/routine word problems 

• Organize or display data 

• Read or produce graphs and tables 

• Execute geometric constructions 


Analyze/Reason 


• Analyze or interpret data 

• Write formal or informal proofs 

• Recognize, generate or create patterns 

• Make generalizations or predictions 

• Identify faulty arguments or misrepresentations of data 

• Reason inductively or deductively 


Solve 

Novel Problems 


• Solve non-routine problems for which students do not have a 
routine strategy or algorithm 

• Design a statistical experiment to study a problem 


Integrate 


• Apply mathematics in real-world situations or to other disciplines 

• Generate, extend, or restate problems 

• Synthesize content and ideas from several sources 



Response Codes for Expectations for Students 

0=No emphasis (Not a performance goal for this topic) 
1=Sllght emphasis (less than 25% of time on this topic) 
2=Moderate emphasis (25%-33% of time on this topic) 
3=Sustalned emphasis (more than 33% of time on this topic) 



2. Science 



Memorize 


• Facts 

• Definitions, Terms 

• Formulas 


Understand 


• Explain concepts 


Concepts 


• Observe and explain teacher demonstrations 

• Explain procedures and methods of science and inquiry 


Perlorm 


• Make observations 


Procedures 


• Collect and record data 

• Use appropriate tools 

• Make measurements, do computations 

• Execute procedure 


Conduct 


• Generate questions, make predictions 


Experiments & 


• Plan and design experiments 


Investigations 


• Test effects of different variables 

• Draw conclusions 

• Communicate investigations & explanations 


Analyze 


• Classify and compare data 


Information 


• Analyze data, recognize patterns 

• Infer from data, draw conclusions 


Apply Concepts 


• Use and integrate concepts 


and Make 


• Apply to real-world situations 


Connections 


• Build or revise theory 

• Make generalizations 



Response Codes for Expectations for Students 

0=No emphasis (Not a performance goal for this topic) 
1=Slight emphasis (less than 25% of time on this topic) 
2=Moderate emphasis (25%-33% of time on this topic) 
3=Sustained emphasis (more than 33% of time on this topic) 
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